Determining quality of experience for communication sessions

ABSTRACT

A computer-implemented method, computer program product, and computing system is provided for determining quality of experience for communication sessions. In an implementation, a method may include determining a plurality of intrusive quality of experience scores associated with a plurality of intrusive audio transmission samples. The method may also include determining a plurality of non-intrusive quality of experience scores associated with a plurality of non-intrusive audio transmission samples. The method may further include deriving a quality of experience coefficient based upon a relationship between the plurality of intrusive quality of experience scores and the plurality of non-intrusive quality of experience scores.

TECHNICAL FIELD

The present disclosure generally relates to systems and method forelectronic communications, and more particularly relates to determiningquality of experience for electronic communication sessions.

BACKGROUND

Various collaboration systems exist for enabling individuals to engagein collaborate activities, such as working together on projects, sharinginformation, and the like. Collaboration systems can be particularlyuseful in allowing geographically dispersed individuals to interact andwork with one another. Using various different collaboration systems, ormulti-functional collaboration systems, individuals who aregeographically remote from one another may, for example, engage incommon work projects, for example, using online team work spaces,participate in interactive discussions, for example, usingteleconferencing or video conferencing systems, and engage in meetings,for example, using electronic meeting systems that allow forpresentations, lectures, seminars, and the like.

SUMMARY OF DISCLOSURE

According to an implementation, a computer-implemented method mayinclude determining a plurality of intrusive quality of experiencescores associated with a plurality of intrusive audio transmissionsamples. The method may also include determining a plurality ofnon-intrusive quality of experience scores associated with a pluralityof non-intrusive audio transmission samples. The method may furtherinclude deriving a quality of experience coefficient based upon arelationship between the plurality of intrusive quality of experiencescores and the plurality of non-intrusive quality of experience scores.

One or more of the following features may be included. Each of theplurality of intrusive quality of experience scores may be based upon,at least in part, a comparison of a transmitted audio signal and areceived audio signal. The plurality of intrusive quality of experiencescores may be determined for a respective plurality of different networkconditions.

Each of the plurality of non-intrusive quality of experience scores maybe based upon, at least in part, identified network conditions. Theplurality of non-intrusive quality of experience scores may bedetermined for a respective plurality of different network conditions.

The derived quality of experience coefficient may be a codec specificcoefficient. The method may further include determining a non-intrusivequality of experience associated with an audio transmission. The codecspecific coefficient may be applied to the non-intrusive quality ofexperience score to obtain a corrected quality of experience metric.

According to another implementation, a computer program product includesa computer readable medium having a plurality of instructions stored onit. When executed by a processor, the instructions may cause theprocessor to perform operations including determining a plurality ofintrusive quality of experience scores associated with a plurality ofintrusive audio transmission samples. Instructions may be include fordetermining a plurality of non-intrusive quality of experience scoresassociated with a plurality of non-intrusive audio transmission samples.Instructions may further be included for deriving a quality ofexperience coefficient based upon a relationship between the pluralityof intrusive quality of experience scores and the plurality ofnon-intrusive quality of experience scores.

One or more of the following features may be included. Each of theplurality of intrusive quality of experience scores may be based upon,at least in part, a comparison of a transmitted audio signal and areceived audio signal. The plurality of intrusive quality of experiencescores may be determined for a respective plurality of different networkconditions.

Each of the plurality of non-intrusive quality of experience scores maybe based upon, at least in part, identified network conditions. Theplurality of non-intrusive quality of experience scores may bedetermined for a respective plurality of different network conditions.

The derived quality of experience coefficient may be a codec specificcoefficient. Instructions may also be included for determining anon-intrusive quality of experience associated with an audiotransmission. The codec specific coefficient may be applied to thenon-intrusive quality of experience score to obtain a corrected qualityof experience metric.

According to another implementation, a computing system includes aprocessor and a memory module coupled with the processor. The processormay be configured for determining a plurality of intrusive quality ofexperience scores associated with a plurality of intrusive audiotransmission samples. The processor may also be configured fordetermining a plurality of non-intrusive quality of experience scoresassociated with a plurality of non-intrusive audio transmission samples.The processor may further be configured for deriving a quality ofexperience coefficient based upon a relationship between the pluralityof intrusive quality of experience scores and the plurality ofnon-intrusive quality of experience scores.

One or more of the following features may be included. Each of theplurality of intrusive quality of experience scores may be based upon,at least in part, a comparison of a transmitted audio signal and areceived audio signal. The plurality of intrusive quality of experiencescores may be determined for a respective plurality of different networkconditions.

Each of the plurality of non-intrusive quality of experience scores maybe based upon, at least in part, identified network conditions. Theplurality of non-intrusive quality of experience scores may bedetermined for a respective plurality of different network conditions.

The derived quality of experience coefficient may be a codec specificcoefficient.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features andadvantages will become apparent from the description, the drawings, andthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view of a distributed computing networkincluding a computing device that executes a quality of experienceprocess according to an implementation of the present disclosure;

FIG. 2 is a flowchart of the quality of experience process of FIG. 1 ,according to an implementation of the present disclosure;

FIG. 3 diagrammatically depicts an implementation of the quality ofexperience process of FIG. 1 , according to an example embodiment;

FIG. 4 diagrammatically depicts an implementation of the quality ofexperience process of FIG. 1 , according to an example embodiment;

FIG. 5 diagrammatically depicts an implementation of the quality ofexperience process of FIG. 1 , according to an example embodiment;

FIG. 6 diagrammatically depicts an implementation of the quality ofexperience process of FIG. 1 , according to an example embodiment;

FIG. 7 is a diagrammatic view of the computing device of FIG. 1 ,according to an implementation of the present disclosure.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Individuals may often engage in communication sessions, utilizing a datanetwork for conveying the communications. Such communication sessionsmay include, for example, Voice-over-IP and Video-over-IPcommunications, in which the communications to and from each party maybe conveyed as packetized data over a data network. The quality of thesetypes of communications can be greatly effected by network conditions.For example, packet loss may be one important factor that may impact theperceived quality of experience of the individuals participating in acommunication session. As packet loss increases the quality of thecommunication may be degraded, for example with the individualsexperiencing broken or garbled audio signals, increased interference andnoise, and the like. Various other network conditions may also impactthe quality of the communication.

Consistent with the present disclosure, the quality of experienceassociated with a media transmission, such as a Voice-over-IP orVideo-over-IP communication, an Internet Protocol TV broadcast, etc.,may be determined. In some implementations, the present disclosure mayallow non-intrusive monitoring methods to be implemented that mayprovide audio quality assessments that may have a degree of accuracythat may generally be similar to audio quality assessments achievablethrough intrusive audio quality measurement assessments. The ability toachieve such relatively high accuracy audio quality assessments throughnon-intrusive monitoring may allow real-time and/or near real-time audioaccuracy assessments to be conducted on an ongoing media transmission,such as a communication session. Real-time and/or near real-time audioquality assessments may allow corrective action, such as adaptivealgorithms, different error correction codes, and the like to beimplemented to provide improved audio quality. In some embodiments, thepresent disclosure may be implemented in connection with ITU(International Telecommunication Union) standardized codec and/or may beimplemented in connection with non-ITU standardized codecs, such asSILK, ISAC, ILBC, SPEEX, etc.

In an embodiment, the present disclosure may allow network conditions,or quality of service factors, such as packet loss, jitter, delay, burstratio, and the like, to the mapped to a quality of experience metricthat may provide an accuracy that may be generally similar to audioquality assessment accuracy achieved using intrusive quality ofexperience assessment. For example, a quality of experience coefficientmay be derived bases upon, at least in part, a mapped relationshipbetween a plurality of quality of experience scores determined using anintrusive assessment technique and a plurality of quality of experiencescores determined using a non-intrusive assessment technique. In anembodiment, the quality of experience coefficient may be applied toquality of experience scores determined using a non-intrusive assessmenttechnique to obtain a corrected quality of experience metric. Thecorrected quality of experience metric may, in some embodiments, have anaccuracy generally similar to a quality of experience metric determinedusing an intrusive assessment technique, but may be obtainable inreal-time and/or near real time.

Referring to FIG. 1 , there is shown quality of experience process 10.For the following discussion, it is intended to be understood thatquality of experience process 10 may be implemented in a variety ofways. For example, quality of experience process 10 may be implementedas a server-side process, a client-side process, or aserver-side/client-side process.

For example, quality of experience process 10 may be implemented as apurely server-side process via quality of experience process 10 s.Alternatively, quality of experience process 10 may be implemented as apurely client-side process via one or more of client-side application 10c 1, client-side application 10 c 2, client-side application 10 c 3, andclient-side application 10 c 4. Alternatively still, quality ofexperience process 10 may be implemented as a server-side/client-sideprocess via server-side quality of experience process 10 s incombination with one or more of client-side application 10 c 1,client-side application 10 c 2, client-side application 10 c 3, andclient-side application 10 c 4. In such an example, at least a portionof the functionality of quality of experience process 10 may beperformed by quality of experience process 10 s and at least a portionof the functionality of quality of experience process 10 may beperformed by one or more of client-side application 10 c 1, 10 c 2, 10 c3, and 10 c 3.

Accordingly, quality of experience process 10 as used in this disclosuremay include any combination of quality of experience process 10 s,client-side application 10 c 1, client-side application 10 c 2,client-side application 10 c 3, and client-side application 10 c 4.

Referring also to FIG. 2 , and as will be discussed in greater detailbelow, quality of experience process 10 may determine 100 a plurality ofintrusive quality of experience scores associated with a plurality ofintrusive audio transmission samples. Quality of experience process 10may also determine 102 a plurality of non-intrusive quality ofexperience scores associated with a plurality of non-intrusive audiotransmission samples. Quality of experience process 10 may furtherderive 104 a quality of experience coefficient based upon a relationshipbetween the plurality of intrusive quality of experience scores and theplurality of non-intrusive quality of experience scores.

Quality of experience process 10 s may be a server application and mayreside on and may be executed by computing device 12, which may beconnected to network 14 (e.g., the Internet or a local area network).Examples of computing device 12 may include, but are not limited to: apersonal computer, a server computer, a series of server computers, amini computer, a mainframe computer, or a dedicated network device.

The instruction sets and subroutines of quality of experience process 10s, which may be stored on storage device 16 coupled to computing device12, may be executed by one or more processors (not shown) and one ormore memory architectures (not shown) included within computing device12. Examples of storage device 16 may include but are not limited to: ahard disk drive; a tape drive; an optical drive; a RAID device; an NASdevice, a Storage Area Network, a random access memory (RAM); aread-only memory (ROM); and all forms of flash memory storage devices.

Network 14 may be connected to one or more secondary networks (e.g.,network 18), examples of which may include but are not limited to: alocal area network; a wide area network; or an intranet, for example.

Examples of client-side applications 10 c 1, 10 c 2, 10 c 3, 10 c 4 mayinclude but are not limited to, one or more of a voice-over-IPapplication, a video-over-IP application, a streaming media application(e.g., capable of receiving and/or rendering streaming media such asstreaming audio and/or streaming video, e.g., IP television, etc.), aweb browser, or a specialized application (e.g., an application runningon a mobile platform). The instruction sets and subroutines ofclient-side application 10 c 1, 10 c 2, 10 c 3, 10 c 4, which may bestored on storage devices 20, 22, 24, 26 (respectively) coupled toclient electronic devices 28, 30, 32, 34 (respectively), may be executedby one or more processors (not shown) and one or more memoryarchitectures (not shown) incorporated into client electronic devices28, 30, 32, 34 (respectively). Examples of storage devices 20, 22, 24,26 may include but are not limited to: hard disk drives; tape drives;optical drives; RAID devices; random access memories (RAM); read-onlymemories (ROM), and all forms of flash memory storage devices.

Examples of client electronic devices 28, 30, 32, 34 may include, butare not limited to, personal computer 28, laptop computer 30, mobilecomputing device 32, notebook computer 34, a netbook computer (notshown), a server computer (not shown), a gaming console (not shown), adata-enabled television console (not shown), and a dedicated networkdevice (not shown). Client electronic devices 28, 30, 32, 34 may eachexecute an operating system.

Users 36, 38, 40, 42 may access quality of experience process 10directly through network 14 or through secondary network 18. Further,quality of experience process 10 may be accessed through secondarynetwork 18 via link line 44.

The various client electronic devices (e.g., client electronic devices28, 30, 32, 34) may be directly or indirectly coupled to network 14 (ornetwork 18). For example, personal computer 28 is shown directly coupledto network 14. Further, laptop computer 30 is shown wirelessly coupledto network 14 via wireless communication channels 46 established betweenlaptop computer 30 and wireless access point (WAP) 48. Similarly, mobilecomputing device 32 is shown wirelessly coupled to network 14 viawireless communication channel 50 established between mobile computingdevice 32 and cellular network/bridge 52, which is shown directlycoupled to network 14. WAP 48 may be, for example, an IEEE 802.11a,802.11b, 802.11g, 802.11n, Wi-Fi, and/or Bluetooth device that iscapable of establishing wireless communication channel 46 between laptopcomputer 30 and WAP 48. Additionally, personal computer 34 is showndirectly coupled to network 18 via a hardwired network connection.

In some embodiments, quality of experience process 10 may communicatewith, interact with, and/or include a component or module of a mediaapplication (e.g., communication application 54). As is generally known,in some embodiments, a media application (e.g., communicationapplication 54) may generally facilitate audio and/or videocommunications between individuals as participants in a communicationsession. For example, communication application 54 may facilitate voiceover IP and/or video over IP communications between communicationsession participants. In various embodiments, a communication sessionmay only include two participants, or may include more than twoparticipants. In some embodiments, communication application 54 mayinclude, and/or may interact with, for example, an electronic meetingapplication, a web conferencing application, or a similar application.An example of communication application 54 may include, but is notlimited to, IBM® Sametime®. (Lotus and Sametime are registeredtrademarks of International Business Machine Corporation in the UnitedStates, other countries, or both).

In other embodiments, a media application may provide streaming mediafor consumption by end users. Examples of streaming media may includestreaming video media and/or streaming audio media. An example ofstreaming media may include, but is not limited to, internet protocoltelevision (e.g., IPTV). Various other forms of streaming media may besimilarly utilized. While the present disclosure may generally be setforth by way of illustrative examples relating to communications betweenparticipants in a communication session, other embodiments involving theprovision and/or consumption of streaming media will be appreciated.

In an embodiment, the instruction sets and subroutines of communicationapplication 54 may be stored, e.g., on storage device 16 associated withserver computer 12, which executes communication application 54, and/oranother suitable storage device. Further, users (e.g., one or more ofusers 36, 38, 40, 42) may access communication application 54 in orderto participate in an electronic communication session (such as a voicecall, a video call, an electronic meeting, or other communicationsession type). The users may access communication application 54 via oneor more suitable applications, such as client side applications 10 c1-10 c 4 (e.g., which may include a voice-over-IP application, avideo-over-IP application, a web browser, a client electronic meetingapplication, or another application) and/or via a different application(not shown). As generally discussed above, a portion and/or all of thefunctionality of quality of experience process 10 may be provided by oneor more of client side applications 10 c 1-10 c 4. For example, in someembodiments quality of experience process 10 (and/or client-sidefunctionality of quality of experience process 10) may be includedwithin and/or interactive with client-side applications 10 c 1-10 c 4,which may include client side electronic communication applications, webbrowsers, or another application. Various additional/alternativeconfigurations may be equally utilized.

As generally discussed above with reference to FIG. 2 , quality ofexperience process 10 determine 100 a plurality of intrusive quality ofexperience scores associated with a plurality of intrusive audiotransmission samples. Quality of experience process 10 may alsodetermine 102 a plurality of non-intrusive quality of experience scoresassociated with a plurality of non-intrusive audio transmission samples.Quality of experience process 10 may further derive 104 a quality ofexperience coefficient based upon a relationship between the pluralityof intrusive quality of experience scores and the plurality ofnon-intrusive quality of experience scores.

As generally discussed above, quality of experience process 10 may allowrelationships between the results of intrusive testing and the result ofnon-intrusive testing to be determined. The relationship may berepresented as a derived quality of experience coefficient that may beutilized to correcting non-intrusive quality of experience assessmentsto provide an accuracy that may, in some implementations, be similar inaccuracy to quality of experience assessments achieved utilizingintrusive quality of experience assessment techniques.

Referring also to FIG. 3 , quality of experience process 10 determine100 a plurality of intrusive quality of experience scores associatedwith a plurality of intrusive audio transmission samples. For example, acommunication session (or other media transmission) may be established,e.g., between a transmitter side (e.g., mobile computing device 32) anda receiver side (e.g., notebook computer 30) through a data network(e.g., network 14). For example, the communication session may include aVoice-over-IP call between user 40 and user 38, via mobile computingdevice 32 and notebook computer 30, respectively. While thecommunication session may include a bidirectional audio transmissions,for the purpose of explanation, mobile computing device 32 may beidentified as a transmitter side device and notebook computer 30 may beidentified as a receiver side device. The communication session may beestablished utilizing a variety of ITU and/or non-ITU codec. Forexample, the communication session may be established using a non-ITUcodec, such as SILK, SPEEX, ILBC, ISAC, or the like.

During the course of the communication session, quality of experienceprocess 10 may determine 100 a plurality of quality of experience scoresassociated with a plurality of intrusive audio transmission samples. Forexample, during the course of the communication session (and/or duringthe course of more than one communication sessions), quality ofexperience process may collect a plurality of intrusive audiotransmission samples. The plurality of intrusive audio transmissionsamples may include audio samples of the communications between user 40and user 38 (e.g., via mobile computing device 32 and notebook computer30, respectively), which may be intrusively analyzed to determine 100 aquality of experience score associated with each of the audiotransmission samples. In an embodiment, each of the plurality ofintrusive audio quality of experience scores may be represented as amean opinion score (MOS), or other score, metric, or value that may berepresentative of a relative quality of experience for an end user(e.g., receiving user 38) associated with each of the intrusive audiotransmission samples.

Each of the plurality of intrusive quality of experience scores may bebased upon, at least in part, a comparison of a transmitted audio signaland a received audio signal. For example, quality of experience process10 may determine 100 each of the plurality of intrusive quality ofexperience scores based upon, at least in part, a degradation betweenthe transmitted audio signal an the received audio signal. For example,as shown in FIG. 3 , during the course of the communication session, anaudio signal may be transmitted from mobile computing device 32 tonotebook computer 30 via network 14. Additionally, the transmitted audiosignal from mobile computing device 32 may be sampled, or recorded, byaudio recorder 150. The recorded signal from mobile computing device 32may be an original, or transmitted signal, e.g., which may be recordedwithout being transmitted through network 14. Additionally, the audiosignal received by notebook computer 30 (e.g., after having beentransmitted through network 14) may also be recorded by audio recorder150. The transmitted audio signal may be degraded as a result of networkconditions, such as packet loss, jitter, latency, burst ratio, etc.,associated with network 14. As such, the audio signal received bynotebook computer 30 may be of a lower quality than the audio singletransmitted from mobile computing device 32. The recorded original, ortransmitted signal, from mobile computing device 32 may be compared, byintrusive analysis 152, to the recorded received audio signal fromnotebook computer 30 to determine a degree of quality change ordegradation between the original audio signal an the received audiosignal. Various metrics, such as MOS, may be utilized to represent theintrusive quality of experience score.

Various algorithms and/or techniques may be utilized by intrusiveanalysis 152 for measuring, or determining, a quality of experiencescore for the transmitted audio received by notebook computer 30. Anexample technique for intrusive analysis of the intrusive audiotransmission samples may include the ITU-T P.862 Perceptual Evaluationof Speech Quality (PESQ) methodology. Other intrusive analysistechniques may equally be utilized. Examples of other such methodologiesmay include, but are not limited to, Perceptual Speech Quality Measure(PSQM), MNB, and PAMS.

The plurality of intrusive quality of experience scores may bedetermined 100 for a respective plurality of different networkconditions. The plurality of different network conditions may include,for example, but are not limited to, different packet loss rate,different jitter, different delay, etc. In an illustrative example,network 14 may include a controllable environment, and the packet lossrate for network 14 may be varied between 0% to 40% in 0.5% increments.An intrusive quality of experience score may be determined 100 for eachpacket loss rate established for network 14, in a manner as generallydescribed above (e.g., recording a transmitted audio signal andrecording a received audio signal, and comparing the transmitted audiosignal with the received audio signal). In other embodiments, e.g., inwhich network 14 may not include a controllable environment, a networkcondition may be determined associated with each determined 100respective intrusive quality of experience score. Over time, thevariation in network conditions may allow intrusive quality ofexperience scores to be determined 100 for a range of networkconditions. Intrusive quality of experience scores may be determined forvariations in network conditions in addition/as an alternative to packetloss rate.

Referring also to FIG. 4 , quality of experience process 10 may alsodetermine 102 a plurality of non-intrusive quality of experience scoresassociated with a plurality of non-intrusive audio transmission samples.In a similar manner as discussed above, a media transmission (such asthe communication session between mobile computing device 32 andnotebook computer 30) may be established via network 14. Quality ofexperience process 10 may determine 102 a plurality of non-intrusivequality of experience scores associated with a plurality of transmissionsamples, for example, associated with a plurality of audio transmissionsamples taken during the course of the communication session betweenuser 40 and user 38. In some embodiments, one or more of the pluralityof non-intrusive quality of experience scores may be determined 102during one or more separate communication sessions, or other mediatransmission instances.

Each of the plurality of non-intrusive quality of experience scores maybe based upon, at least in part, identified network conditions. Forexample, for each non-intrusive transmission sample, one or more networkconditions may be determined, e.g., by non-intrusive analysis 154.Non-intrusive analysis 154 may utilized one or more non-intrusivetechniques or algorithms to determine 102 each of the plurality ofnon-intrusive quality of experience scores. An example of anon-intrusive quality of experience technique may include the ITU-TG.107 method, which may also be referred to as the E-model method. Insome implementations, non-intrusive quality of experience techniques maygenerally provide measures of quality of service associated with thenetwork, which may not always be accurate indicators of actual qualityof experience realized by an end user (such as user 38, as a receivingparticipant of a portion of a communication session). Othernon-intrusive quality of experience methodologies based upon, at leastin part, one or more network conditions, may equally be utilized.Examples of network conditions that may be analyzed for determining 102the plurality of non-intrusive quality of experience scores may include,but are not limited, packet loss rate, jitter, delay, etc., associatedwith network 14.

The plurality of non-intrusive quality of experience scores may bedetermined for a respective plurality of different network conditions.The plurality of different network conditions may include, for example,but are not limited to, different packet loss rate, different jitter,different delay, etc. In a similar illustrative example as describedabove, network 14 may include a controllable environment, and the packetloss rate for network 14 may be varied between 0% to 40% in 0.5%increments. A non-intrusive quality of experience score may bedetermined 102 for each packet loss rate established for network 14, ina manner as generally described above (e.g., determining one or morenetwork conditions and implementing a non-intrusive analysis technique).In other embodiments, e.g., in which network 14 may not include acontrollable environment, a network condition may be determinedassociated with each determined 102 respective non-intrusive quality ofexperience score. Over time, the variation in network conditions mayallow non-intrusive quality of experience scores to be determined 102for a range of network conditions. Non-intrusive quality of experiencescores may be determined for variations in network conditions inaddition/as an alternative to packet loss rate.

The plurality of non-intrusive audio transmission samples may beobtained during the same communication session, or during a differentcommunication session, than the plurality of intrusive audiotransmission samples. In an embodiment in which the plurality ofnon-intrusive audio transmission samples may be obtained during the samecommunication session as the plurality of intrusive audio transmissionsamples, each non-intrusive audio transmission sample may be obtained ata generally similar time frame as a corresponding intrusive audiotransmission sample. Other configurations may equally be implemented.

Quality of experience process 10 may further derive 104 a quality ofexperience coefficient based upon a relationship between the pluralityof intrusive quality of experience scores and the plurality ofnon-intrusive quality of experience scores. For example, and referringalso to FIG. 5 , quality of experience process 10 may derive 104 therelationship between the quality of experience score determined 100 byintrusive analysis 152 and the quality of experience score determined102 by non-intrusive analysis (e.g., which may provide the main networkquality of service factors that may effect quality of experience) forthe different network conditions. In an embodiment, deriving 104 therelation between the intrusive testing and the non-intrusive testing mayutilize curve fitting techniques, such as by an approximation of anon-linear regression model.

The derived quality of experience coefficient may be a codec specificcoefficient. Continuing with the above example in which a quality ofexperience associated with a communication session may be determined100, 102, as discussed above the communication session may beestablished utilizing a given codec (e.g., an ITU codec and/or a non-ITUcodec, such as SILK, SPEEX, ILBC, etc.). Accordingly, the plurality ofintrusive quality of experience scores and the plurality ofnon-intrusive quality of experience scores may be relevant to the codecutilized for the communication session. Further, the derived 104 qualityof experience coefficient may be relevant to the same codec, giving riseto a codec specific coefficient. Accordingly, the same process asgenerally described above may be carried out for a variety of differentcodecs, thereby providing a corresponding variety of codec specificcoefficients.

Quality of experience process 10 may allow the derived quality ofexperience coefficient to be utilized determine the quality ofexperience associated with streaming media, such transmitted audio of acommunication session. As generally described above, in someimplementations quality of experience process 10 may also the quality ofexperience associated with streaming media to be determined in real-timeand/or near real-time. In an illustrative example, as shown in FIG. 6 ,users 38 and 40 may participate in a communication session, e.g., vianotebook computer 30 and mobile computing device 32 respectively.Quality of experience process 10 may allow a quality of experience, asobserved by an end user (e.g., a participant in a communication session,such as user 38 and/or user 40) to be determined at any point in timeduring the communication session. For example, quality of experienceprocess 10 may determine 106 a non-intrusive quality of experienceassociated with an audio transmission, e.g., via non-intrusive analysis154. Quality of experience process 10 may determine 106 thenon-intrusive quality of experience intermittently and/or periodically.An interval for intermittently determining 106 the non-intrusive qualityof experience associated with the communication session may bedetermined, for example, to provide relatively low additional demands onnetwork 14. As such, determining 106 the non-intrusive quality ofexperience associated with the communication session may typically notnegatively impact network 14, which could thereby degrade the quality ofexperience associated with the communication session. For example,quality of experience process 10 may determine 106 the non-intrusivequality of experience associated with the communication session every1-20 seconds. It will be appreciated that other time intervals mayequally be utilized.

Quality of experience process 10 may apply 108 the codec specificcoefficient to the non-intrusive quality of experience score to obtain acorrected quality of experience metric. The codec specific coefficientmay correspond to the codec that is utilized for conducing thecommunication session between user 38 and user 40. As such, a codecspecific coefficient may be applied 108 that may represent therelationship between a non-intrusive quality of experience scoreassociated with the communication session (which may utilize thespecific codec) and a quality of experience that may be observed by anend user. As a result of applying 108 the codec specific coefficient tothe determined 106 non-intrusive quality of experience score, acorrected quality of experience metric may be provided, which mayrelatively accurately indicate a quality of experience for thecommunication session as may be observed by a participant in thecommunication session (e.g., by user 38 and user 40 in the illustratedembodiment).

Accordingly, in some embodiments, quality of experience process 10 mayprovide a single metric to quantify a quality of experience associatedwith streaming media, such as audio in an communication session, or thelike. The single metric may be non-intrusively obtained, and may providea real-time and/or near real-time measure of the quality of experience,e.g., as may be observed by an end user, such as a participant in acommunication session (e.g., user 38, in the illustrative example). Thesingle metric may allow, for example, real-time and/or near real-timedynamic resource management and/or tuning of the network and/orcommunication session attributes (such as error correction, etc.) duringthe course of a communication session. In some embodiments, quality ofexperience process 10 may be implemented as a plug-in, or feature of, asoftphone application, or other communication application.

Referring also to FIG. 7 , there is shown a diagrammatic view ofcomputing system 12. While computing system 12 is shown in this figure,this is for illustrative purposes only and is not intended to be alimitation of this disclosure, as other configuration are possible. Forexample, any computing device capable of executing, in whole or in part,quality of experience process 10 may be substituted for computing device12 within FIG. 7 , examples of which may include but are not limited toclient electronic devices 28, 30, 32, 34.

Computing system 12 may include microprocessor 200 configured to e.g.,process data and execute instructions/code for quality of experienceprocess 10. Microprocessor 200 may be coupled to storage device 16. Asdiscussed above, examples of storage device 16 may include but are notlimited to: a hard disk drive; a tape drive; an optical drive; a RAIDdevice; an NAS device, a Storage Area Network, a random access memory(RAM); a read-only memory (ROM); and all forms of flash memory storagedevices. IO controller 202 may be configured to couple microprocessor200 with various devices, such as keyboard 204, mouse 206, USB ports(not shown), and printer ports (not shown). Display adaptor 208 may beconfigured to couple display 210 (e.g., a CRT or LCD monitor) withmicroprocessor 200, while network adapter 212 (e.g., an Ethernetadapter) may be configured to couple microprocessor 200 to network 14(e.g., the Internet or a local area network).

As will be appreciated by one skilled in the art, the present disclosuremay be embodied as a method (e.g., executing in whole or in part oncomputing device 12), a system (e.g., computing device 12), or acomputer program product (e.g., encoded within storage device 16).Accordingly, the present disclosure may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present disclosure may take the form of a computer program producton a computer-usable storage medium (e.g., storage device 16) havingcomputer-usable program code embodied in the medium.

Any suitable computer usable or computer readable medium (e.g., storagedevice 16) may be utilized. The computer-usable or computer-readablemedium may be, for example but is not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, device, or propagation medium. More specific examples (anon-exhaustive list) of the computer-readable medium may include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, a portable compact disc read-onlymemory (CD-ROM), an optical storage device, a transmission media such asthose supporting the Internet or an intranet, or a magnetic storagedevice. The computer-usable or computer-readable medium may also bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave. The computer usable program code may betransmitted using any appropriate medium, including but not limited tothe Internet, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentdisclosure may be written in an object oriented programming languagesuch as Java, C#.NET, PHP, C++ or the like. However, the computerprogram code for carrying out operations of the present disclosure mayalso be written in conventional procedural programming languages, suchas the “C” programming language or similar programming languages. Theprogram code may execute entirely on the user's computer, partly on theuser's computer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through a local area network/a widearea network/the Internet (e.g., network 14).

The present disclosure is described with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the disclosure. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, may be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor (e.g., processor 200) of a general purposecomputer/special purpose computer/other programmable data processingapparatus (e.g., computing device 12), such that the instructions, whichexecute via the processor (e.g., processor 200) of the computer or otherprogrammable data processing apparatus, create means for implementingthe functions/acts specified in the flowchart and/or block diagram blockor blocks.

These computer program instructions may also be stored in acomputer-readable memory (e.g., storage device 16) that may direct acomputer (e.g., computing device 12) or other programmable dataprocessing apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture including instruction means which implement thefunction/act specified in the flowchart and/or block diagram block orblocks.

The computer program instructions may also be loaded onto a computer(e.g., computing device 12) or other programmable data processingapparatus to cause a series of operational steps to be performed on thecomputer or other programmable apparatus to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide steps for implementingthe functions/acts specified in the flowchart and/or block diagram blockor blocks.

The flowcharts and block diagrams in the figures may illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustrations,and combinations of blocks in the block diagrams and/or flowchartillustrations, may be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the disclosure. Theembodiment was chosen and described in order to best explain theprinciples of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

Having thus described the disclosure of the present application indetail and by reference to embodiments thereof, it will be apparent thatmodifications and variations are possible without departing from thescope of the disclosure defined in the appended claims.

1. A computer-implemented method comprising: determining, by a processor, a plurality of intrusive quality of experience scores associated with a plurality of intrusive audio transmission samples during a communication session, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions including different packet loss rates up to 40% in increments of 0.5%; determining, by the processor, a plurality of non-intrusive quality of experience scores associated with a plurality of non-intrusive audio transmission samples, wherein each non-intrusive audio transmission sample of the plurality of non-intrusive audio transmission samples is obtained at a same time frame during the communication session as a corresponding intrusive audio transmission sample of the plurality of intrusive audio transmission samples; and deriving, by the processor, a quality of experience coefficient based upon a relationship between the plurality of intrusive quality of experience scores and the plurality of non-intrusive quality of experience scores, wherein the derived quality of experience coefficient is a codec specific coefficient.
 2. The computer-implemented method of claim 1, wherein each of the plurality of intrusive quality of experience scores are based upon, at least in part, a comparison of a transmitted audio signal and a received audio signal.
 3. The computer-implemented method of claim 1, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 4. The computer-implemented method of claim 1, wherein each of the plurality of non-intrusive quality of experience scores are based upon, at least in part, identified network conditions.
 5. The computer-implemented method of claim 1, wherein the plurality of non-intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 6. (canceled)
 7. The computer-implemented method of claim 1, further including determining a non-intrusive quality of experience associated with an audio transmission and applying the codec specific coefficient to the non-intrusive quality of experience score to obtain a corrected quality of experience metric.
 8. A computer program product comprising a computer readable medium having a plurality of instructions stored thereon, which, when executed by a processor, cause the processor to perform operations including: determining a plurality of intrusive quality of experience scores associated with a plurality of intrusive audio transmission samples during a communication session, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions including different packet loss rates up to 40% in increments of 0.5%; determining a plurality of non-intrusive quality of experience scores associated with a plurality of non-intrusive audio transmission samples, wherein each non-intrusive audio transmission sample of the plurality of non-intrusive audio transmission samples is obtained at a same time frame during the communication session as a corresponding intrusive audio transmission sample of the plurality of intrusive audio transmission samples; and deriving a quality of experience coefficient based upon a relationship between the plurality of intrusive quality of experience scores and the plurality of non-intrusive quality of experience scores, wherein the derived quality of experience coefficient is a codec specific coefficient.
 9. The computer program product of claim 8, wherein each of the plurality of intrusive quality of experience scores are based upon, at least in part, a comparison of a transmitted audio signal and a received audio signal.
 10. The computer program product of claim 8, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 11. The computer program product of claim 8, wherein each of the plurality of non-intrusive quality of experience scores are based upon, at least in part, identified network conditions.
 12. The computer program product of claim 8, wherein the plurality of non-intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 13. (canceled)
 14. The computer program product of claim 8, further including instructions for determining a non-intrusive quality of experience associated with an audio transmission and applying the codec specific coefficient to the non-intrusive quality of experience score to obtain a corrected quality of experience metric.
 15. A computing system comprising: a processor and a memory module coupled with the processor, the processor being configured for: determining a plurality of intrusive quality of experience scores associated with a plurality of intrusive audio transmission samples during a communication session, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions including different packet loss rates up to 40% in increments of 0.5%; determining a plurality of non-intrusive quality of experience scores associated with a plurality of non-intrusive audio transmission samples, wherein each non-intrusive audio transmission sample of the plurality of non-intrusive audio transmission samples is obtained at a same time frame during the communication session as a corresponding intrusive audio transmission sample of the plurality of intrusive audio transmission samples; and deriving a quality of experience coefficient based upon a relationship between the plurality of intrusive quality of experience scores and the plurality of non-intrusive quality of experience scores, wherein the derived quality of experience coefficient is a codec specific coefficient.
 16. The computing system of claim 15, wherein each of the plurality of intrusive quality of experience scores are based upon, at least in part, a comparison of a transmitted audio signal and a received audio signal.
 17. The computing system of claim 15, wherein the plurality of intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 18. The computing system of claim 15, wherein each of the plurality of non-intrusive quality of experience scores are based upon, at least in part, identified network conditions.
 19. The computing system of claim 15, wherein the plurality of non-intrusive quality of experience scores are determined for a respective plurality of different network conditions.
 20. (canceled)
 21. The computing system of claim 15, further including determining a non-intrusive quality of experience associated with an audio transmission and applying the codec specific coefficient to the non-intrusive quality of experience score to obtain a corrected quality of experience metric. 